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ABSTRACT 

This paper describes the AlChE Concept Warehouse, a recently developed web-based instructional 
tool that enables faculty within the discipline of chemical engineering to better provide their students 
concept-based instruction. It currently houses over 2,000 concept questions and 10 concept invento¬ 
ries pertinent to courses throughout the core chemical engineering curriculum. These questions are 
available for faculty use both as in-class concept-based clicker questions (or ConcepTests) and stand¬ 
alone concept inventories, and can be accessed in various formats (online or offline) for use in class 
and on assignments and exams. The design philosophy is to make the tool versatile so that it can be 
used in the way that best fits with the instructor’s teaching philosophy and the program’s educational 
environment. Instructors and students perceive it to catalyze engagement and promote learning. While 


WINTER 2014 


1 




ADVANCES IN ENGINEERING EDUCATION 

The AlChE Concept Warehouse 


domain specificity is critical to the targeted development of tools like the one we describe, we argue 
that the computer-based approach is generic and could be applied to any engineering discipline. One 
objective in describing this tool and characterizing our experiences using it is to contribute to such 
wider adaptation. 

Key Words: educational technology; interactive learning environments; teaching/learning strategies 


INTRODUCTION 


Overview 

Many engineering educators and industry partners emphasize the need for students to apply 
their knowledge to new and challenging problems ( Educating the Engineer of 2020: Adapting 
Engineering Education to the New Century, 2005). In order to do so, students must learn with 
understanding (Bransford, Brown, & Cocking, 1999). A lack of conceptual understanding has 
been shown to severely restrict students’ ability to solve new problems, since they do not have 
the foundational understanding to use their knowledge in new situations (Hestenes, Wells, & 
Swackhamer, 1992). However, science and engineering classrooms often reward students more 
for rote learning than for conceptual understanding (Elby, 1999; Felder & Brent, 2005). There is 
clearly a need for more emphasis on conceptual understanding. In this paper, we refer to concept- 
based instruction as the use of pedagogies and corresponding learning activities whose primary 
objectives are to make students aware of their need for conceptual understanding and then foster 
that understanding. 

The effectiveness of many forms of concept-based instruction depends critically on the 
availability of high-quality concept questions. These questions can be time-consuming and difficult 
to construct, posing one of the biggest barriers to implementing this type of pedagogy (Beatty, 
Gerace, Leonar, & Dufresne, 2006; Crouch, Watkins, Fagen, & Mazur, 2007; Fagen, Crouch, & Mazur, 
2002). As Kay & LaSage (2009, p. 824) state in their review of the recent literature, “Unfortunately 
there are very few collections of ARS (i.e, audience response system or clicker) questions avail¬ 
able in most fields, so instructors have to develop original questions, a process that is very time 
consuming.” 

The AlChE Concept Warehouse is a web-based instructional tool that decreases this barrier by 
housing concept questions pertinent to a specific discipline - chemical engineering (ChE). This 
cyber-enabled infrastructure is maintained through the Education Division of the discipline’s primary 
professional society, the American institute of Chemical Engineers (AlChE). Questions are available 
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for faculty use both as in-class ConcepTests and stand-alone concept inventories, and can be used 
in many ways as best fits with the instructor’s teaching philosophy and the programs educational 
environment. 

In this paper, we provide an overview that describes the software architecture of the AlChE 
Concept Warehouse, report on the initial community use of this tool, and investigate instructor and 
student perceptions about its use. While domain specificity is critical to the targeted development 
of tools like the one we describe, we argue that the computer-based approach is generic and could 
be applied to any STEM discipline or community. One objective in describing this tool and charac¬ 
terizing our experiences in using it is to help enable such wider adaption. 

Concept-based Pedagogy 

The pedagogical methods and instruments to promote concept-based instruction that our tool 
employs are based on those developed by and studied in the physics education research community 
for three decades. Two concept-based pedagogical methods and instruments have dramatically 
reshaped how conceptual teaching and learning are viewed in college physics classrooms: Con¬ 
cepTests (Mazur, 1997) and concept inventories (Hestenes, Wells, & Swackhamer, 1992). Both require 
high quality concept questions in order to be effective. High quality concept questions are typically 
(but not necessarily) multiple choice, conceptually challenging, and require little to no calculation 
so students cannot rely on equations to reach an answer. In addition, they contain incorrect choices, 
termed distracters, which are attractive to students because they align with common misconceptions. 
In the following sections, we provide brief descriptions of ConcepTests and concept inventories. 

ConcepTests and Active Learning 

In his book Peer Instruction, Eric Mazur (1997) describes the use of ConcepTests to engage stu¬ 
dents in conceptual learning during lectures. Mazur developed an active learning pedagogy which 
he terms peer instruction as a way of delivering ConcepTests in class. We illustrate the use of Con¬ 
cepTests with this pedagogy; however, there are many other ways that instructors have actively 
incorporated them in concept-based instruction (Beatty & Gerace, 2009; Dufresne, Gerace, Leon¬ 
ard, Mestre, & Wenk, 1996; Kalman, Milner-Bolotin, & Antimirova, 2010; Nicol & Boyle, 2003). Peer 
instruction includes structured questioning in which the instructor poses a ConcepTest and then 
all students respond independently, using clickers, smart phones, laptops, or flash cards. Students 
then discuss their answers with one another in small groups and respond again individually. Peer 
instruction encourages students to reflect on conceptual problems, think through the arguments 
being developed, and put them into their own words. It also provides both student and instructor 
with feedback regarding student understanding of the concept being tested. 
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Two separate studies, each with over 5,000 student participants, have found that classes 
using active learning methods such as peer instruction had on average double the conceptual 
learning gains (Hake, 1998) and a 25% higher pass rate (Poulis, Massen, Robens, & Gilbert, 
1998) than traditional lecture classes. Deslauriers, Schelew, & Wieman (2011) compared the 
performance of two sections of students studying electromagnetic waves; the section taught 
by an inexperienced postdoctoral fellow using active learning techniques which focused on 
concept-based instruction outperformed, by 2.5 standard deviations, the section taught by an 
experienced professor with a record of high evaluations. Clearly, tools that enable faculty to 
implement these types of methods show great promise for positively impacting student learn¬ 
ing in the classroom. 

Concept Inventories 

Whereas ConcepTests are used to promote student engagement and enable formative assess¬ 
ment, concept inventories provide a summative assessment tool to evaluate and demonstrate the 
effectiveness of active learning and other pedagogies. Concept inventories are valid and reliable 
instruments that consist of high quality concept questions. Validity is a measure of how well the 
concept inventory measures the intended concepts, as evaluated by experts, student observations, or 
other means within a single population of students or a variety of populations (Nelson, Geist, Miller, 
Streveler, & Olds, 2007). Reliability is a measure of the degree to which repeated administrations 
of the concept inventory produce the same results (Nelson, Geist, Miller, Streveler, & Olds, 2007). 
Concept inventories are used as an objective pre/post measure of an instructional intervention and 
have been used to inform instruction by identifying student misconceptions. 

The seminal concept inventory, the Force Concept Inventory (FCI), provided an instrument 
to measure students’ fundamental conceptual understanding of Newtonian mechanics (Halloun, 
1985; Hestenes, Wells, & Swackhamer, 1992). Since development of the FCI, concept inventories 
have been created in a variety of engineering subjects, including statics (Steif & Dantlzer, 2005), 
dynamics (Gray, et al., 2005), and fluid mechanics (Martin, Mitchell, & Newell, 2004). Recently, 
a web-site has been developed to house concept inventories throughout engineering (Imbrie & 
Reed-Rhoads, 2011). The set of concept inventories pertinent to chemical engineering include: 
the Thermal and Transport Concept Inventory (TTCI) (Streveler, et al., 2011; Streveler, Olds, Miller, 
& Nelson, 2003), the Heat and Energy Concept Inventory (Prince, Vigeant, & Nottis, 2012), the 
Materials Concept Inventory (Krause, Decker, & Griffin, 2003), the Engineering Thermodynamics 
Concept Inventory (Midkiff, Litzinger, & Evans, 2001), the Thermodynamics Concept Inventory 
(Vigeant, Prince, & Nottis, 2011b), and a preliminary Material and Energy Balances Concept Inven¬ 
tory (Shallcross, 2010). 
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ARCHITECTURE 


Overview 

The AlChE Concept Warehouse has been developed to provide instructors and their students 
with a cyber-enabled infrastructure housing conceptual questions that contain content spanning the 
core chemical engineering curriculum. As of October 2013, approximately 2,000 concept questions 
(ConcepTests) and 10 valid and reliable concept inventories were available for searching, viewing, 
and using in courses. Instructor and student interfaces (discussed below) are available for the com¬ 
munity at l http://cw.edudiv.orgl, and university faculty can obtain an account through this site. 

Design Principles 

The design architecture is based on the first and third authors’ experience developing a similar 
system that was institution specific (Koretsky & Brooks, 2008). The challenge we faced in extension to 
the chemical engineering community was to create an architecture that would be versatile and could 
serve faculty from many institutions who have a wide range of philosophies, climates, and needs. In 
order to accomplish this objective, we based design on several overarching principles including: 

• Question Quality Principle: Provide faculty a resource of high quality conceptual questions, 
both as ConcepTests and concept inventories (as available). 

• Question Quantity Principle: Provide a large enough question pool so that instructors can find 
questions pertaining to the specific concept they are teaching. 

• Emergent Use Principle: Be versatile in how questions can be deployed in instruction so that 
instructors can use it in ways that best fit their philosophy and context and these ways can 
change with increasing participation. 

• Familiarity Principle: As much as possible, design the layout of the website to be intuitive and 
match other common web sites with which users may be familiar. 

• Support Principle: Provide multiple ways to technically support faculty who adopt the tool. 
Such support can be online resources, webinars, workshops, and email. 

• Community Contribution Principle: Provide a way for faculty to contribute their own materials 
and to participate as peer reviewers of content. 

• Data Collection Principle: Collect question response data for instructors to use in class and 
to provide empirical evidence to characterize questions and identify student misconceptions 
(with informed consent). 

Modes of Use 

The intent of the AlChE Concept Warehouse is to provide a tool that instructors can use that best fits 
with their teaching philosophy and learning environment. In essence, we want to cultivate emergent uses 
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Mode of Use 

Description 

Offline 


Download questions via Microsoft Word or PowerPoint to use on 
homework, tests, or with external clicker systems 

Online* 

Homework 

Used online outside of class through student interface as homework 


In-Class, with Laptops or 
Cell Phones 

Used in class through web browser allowing short answer 
explanations and confidence follow-up 


In-Class, with Turning 
Point Clickers 

Used in class with Turning Point clickers using the AlChE Concept 
Warehouse java applet 


* requires students to interface with the site 

Table 1. Predicted modes of use of the AlChE Concept Warehouse and their description 


in which faculty leverage and build their own knowledge and experience rather than to attempt to pre¬ 
scribe a unitary, specific way to implement concept-based instruction (Henderson, Beach, & Finkelstein, 
2011). To this end, the tool has been designed to be versatile with four general ways that instructors can 
use it for instruction (Emergent Use Principle). These four modes of use are supported through Quick 
Start Guides written as step-by-step instructions and available through the instructor interface. In addi¬ 
tion there are video walkthroughs for a subset of user activities within the AlChE Concept Warehouse. 
These resources are intended for new users to facilitate their initial use (Support Principle). 

Table 1 provides a summary of these modes of use. At all levels of use, instructors can create a 
“class” which dedicates workspace for a particular course that they are teaching. Within the class 
they can select a set of ConcepTest questions to create a “test,” which is a collection of questions 
to be used in a single class meeting or out of class assignment. Also they can assign concept inven¬ 
tories to the students in the class. 

Offline Use 

Offline refers to faculty downloading questions, either as a Microsoft Word document or Power¬ 
Point slides rather than using the web based infrastructure directly to deliver the questions to stu¬ 
dents. Examples of offline use include: use on a homework set, test, quiz, or in class with an external 
clicker system. This form of use does not require students to access the site. Even at the basic level 
of using offline, instructors already using peer instruction or active learning with concept questions 
need only make minor changes to current practices and the AlChE Concept Warehouse can save 
them preparation time by providing easy access to high quality concept questions. 

Online Use 

If an instructor wants to use more of the features available Online, instead of downloading ques¬ 
tions they can integrate the use of clickers or have students log in and answer ConcepTests and 
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concept inventories on their laptops or smart phones (either “live” in-class or for homework). A 
benefit of this mode is that it enables instructors to view results in an aggregated and tabulated 
format. The results may be archived for later use and can be downloaded in Microsoft Excel format. 
If instructors solicit responses via laptops or smartphones, they can also prompt students for short 
answer explanations of their answer choice and ask them to rate their confidence. Such written 
reflection is perceived by students as helpful to learning (Koretsky & Brooks, 2012). These more- 
involved features require students to interface with the website. 

Online use also benefits the education research community since usage data can be used to in¬ 
form ConcepTest and concept inventory development (Data Collection Principle). Only aggregate, 
anonymous data are used, taken from students who agree to participate as acknowledged through 
their acceptance of the IRB-approved informed consent available on the site. For example, data from 
student selections could be used for item testing, a critical step in concept inventory development. 
This synergy will allow the question pool for use in concept inventories to greatly expand. 

Instructor Interface 

The instructor interface is organized into seven main sections, accessible by their corresponding 
tabs: Home, ConcepTests, Concept Inventories, Instructional Tools, Classes, Profile, and Support. 
Table 2 provides a description of the functions available to instructors, organized by submenu sec¬ 
tions within each tab. 

In order to maximize compatibility and minimize complexity, an effort was made to design the instruc¬ 
tor interface to match with the current web experiences of potential adopters, to be similar to other 
web-based interfaces, and to be user-friendly (Familiarity Principle). One approach to accomplishing 
this design objective was to predict and accommodate different ways users might leverage the AlChE 
Concept Warehouse in their instruction as previously summarized in Table 1. As an example, we present 
a screen shot of the ConcepTest search tab that enables instructors to find questions in Figure 1. 

In addition, instructors may actively contribute by adding their own questions to the database 
through a scaffolded web tab, as partially shown in Figure 2 (Community Contribution Principle). 
Instructors can use any of the questions that they develop solely in their own classes or, alternatively, 
they can choose to submit a question they have developed for use by the entire community. In the 
latter case, the questions are incorporated pending peer review (Quality Question Principle). They 
can create multiple-choice, multiple correct multiple-choice, ranking, and short answer questions. 
These questions can then be associated with classes, the misconceptions they address, textbooks, 
topics, and sub-topics. Comments can also be left for instructors when they are searching for or 
deciding on using questions. These comments can include information about the correct answer, a 
distractor, or the use of the question. 
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Tab 

Submenu 

Sections 

Description of Functions 

<D 

B 

o 


View a brief summary of the latest changes, including: highlights of AIChE Concept Warehouse news, 
added questions, new tutorials and comments about submitted questions. 


Search 

View, filter, and search for questions. Then, select question(s) for use in class. 


Manage 

Tests 

Organize, group, download (MS Power Point, MS Word), or assign (via projection in-class or sent to 
student laptops or smartphones) ConcepTests. Confidence and short answer explanation prompts can 
be added to questions during assignment. 

£ 

& 

o 

« 

o 

U 

Statistics 

View information after questions have been answered, including all or a subset of options: question 
title and text, question images, percent correct, answer choice distribution, average confidence, 
number of students answered, and pedagogy recommendation. 


New 

Question 

Add new questions. Question information can include: question title, question and answer images, 
answer options, comments for faculty, and applicable research data. Question types include: multiple 
choice - single right answer or multiple right answer, short answer, and ranking. Questions can also 
be tagged by class, misconception, topic, and textbook. 

0) 

'Ch 

o 

Browse 

View available concept inventories and select for use in class. Additional concept inventory 
information is available for viewing, such as: research data, development history, list of individual 
questions and an answer key. 

§ 

Oh 

<D 

o 

Manage 

Inventories 

Assign concept inventories (via projection in-class or sent to student laptops or smartphones), either 
in complete form or subsections. Confidence and short answer explanation prompts can be added to 
questions during assignment. 

a 

o 

U 

Statistics 

View information after concept inventory questions have been answered, including all or a subset of 
options: question name, question text, question image, percent correct, answer distribution, average 
confidence, and number of students answered. 


Class List 

Create and delete personal classes and associate personal classes with general classes. 

Classes 

Manage 

Class 

Manage the class roster and grade sheet. Add students to or remove them from the class roster as well 
as send students an email instructing them on how to set up an account. View and download the grade 
sheet, which includes student responses to questions (correct/incorrect, written responses, written 
explanations, confidence) 


Preferences 

Set personal preferences such as: show or hide tooltips, show or hide answer option comments, and 
show or hide the correct answer indicator in question previews. 

o 

Demographic 

Information 

Report institution, schedule type, approximate first year teaching, and approximate first year using 
active learning. 

£ 

My Clicker 

Select clicker type, download clicker integration application, and register clicker receiver. 


User 

Agreement 

Displays the end user license agreement (EULA). Accepting the EULA is a required prerequisite to 
use of the AIChE Concept Warehouse and is displayed upon initial log-in. 


Webinars 

Register for webinars offered by collaborators (e.g. Getting Started) 


Quick Start 

Step-by-step instructions available online or as a download (PDF) for the four modes of use. Each mode 
of use is summarized to help faculty determine which mode best suits their current teaching philosophy. 

tn 

o 

Intro to 
Pedagogy 

View a collection of journal articles and videos to facilitate the integration of concept-based 
pedagogy into their classes. 

1 

00 

Tutorials 

Watch videos to help instructors use the AIChE Concept Warehouse. 


Chat Forum 

Interact with other community members through a bulletin board. This includes a forum for 
frequently asked questions (e.g How do I search for concept questions?) 


Helpful Links 

Links to external resources on concept-based pedagogy, and active learning. 


News Archive 

View the full history of the news about the AIChE Concept Warehouse. 

Table 2. Summary of instructor interface functions organized by sub menu sections 

within each tab 
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HOME V CONCEPTESTS V CONCEPT V INSTRUCTIONAL V CLASSES V PROFILE V SUPPORT M 
_ INVENTORIES | TOOLS g _I_I_■ 

ABOUT SEARCH MANAGE TESTS STATISTICS NEW QUESTION 


Class : default 
Classes 


□ Energy Balances 

□ Engineering 
Orientation 

□ Fluid Dynamics 

□ General Chemistry 

□ Heat Transfer 

□ Mass Transfer 

□ Material Balances 

□ Materials Science 

□ other 

□ Reaction 
Engineering 

EZl Thermodynamics 

□ Workshop 

K3 Misconceptions 


d Topics 


d Surface Features 


ED My Questions 


d Unused (self) 


d Type 


1 Suggestions [ 


Figure 1. Screenshot of the ConcepTest search tab in the instructor interface of the AlChE 
Concept Warehouse 


Search Reset Advanced Search 


Showing "il-lOO^ of 792 results (in Os) 

x — n 


Results per page: 100 ▼ 


0 ID:1 Modified Throttling Valve 
——■$]—^ 


An ideal gas flows steadily through the piping system and 
valve shown below. The inlet pressure and temperature are 
Pi and T... 


Contributor: Bill Brooks 
AuthorRon Miller 
Type: Multiple Choice 
Rating 


Class: Energy Balances 
Thermodynamics 


0 ID:13 Thermite Reaction 



Consider a completely insulated, closed system of fixed 
volume. The exothermic thermite reaction takes place in the 
system. What happens to the system's energy? 

Contributor: Bill Brooks Class: Energy Balances 

AuthonJohn Falconer Thermodynamics 

Type: Multiple Choice 
Rating 

***** 


□ ID: 14 Exothermic Reaction in 
Adiabatic SS Flow Reactor 


The exothermic reaction A -> B reaches an equilibrium 
conversion of 25% in an adiabatic steady state flow reactor. 
Which of the following will likely increase conversion? 
Contributor: John Falconer Class: Thermodynamics 

Type: Multiple Choice Reaction Engineering 

Rating 


Instructors who use the tool in the Online mode, have access to student performance data as shown 
in Figure 3. An instructor receives summary data for all questions assigned in a class (as shown in 
the main tab), and has historical data and can retrieve student response data from previous classes. 
In addition, instructors can access student explanations in two ways, as shown in the insets. First, 
the entire response set is available to the instructor as a word cloud. Second, a sortable, anonymous 
list of individual written responses for each question is available together with the multiple-choice 
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HOME W CONCEPTESTS V CONCEPT W INSTRUCTIONAL V CLASSES 

. INVENTORIES I TOOLS I 



ABOUT SEARCH MANAGE TESTS STATISTICS NEW QUESTION 


YOUR QUESTION 


Title : Modified Throttling Valve 

General Question Text: 

p Equation Editor 


U A* A x 


An ideal gas flows steadily through the piping system and valve 


Question Image (optional - gif, png, or jpg) 

I Browse... I No file selected. 


Question Type: 

® Multiple Choice - single right answer 
O Multiple Choice - mutiple right answers 
Q Short Answer 
O Ranking 


Figure 2. Screenshot of the question input tab in the instructor interface of the AlChE 
Concept Warehouse 


response selected and the selected confidence rating. This data can provide instructors rich infor¬ 
mation about the level and distribution of their students’ conceptual understanding. 

Student Interface 

The student interface is simpler than the instructor interface and has only three tabs: Home, Ques¬ 
tions, and Profile. The tabs, submenu sections, and function descriptions are presented in Table 3. 
The questions tab, shown in Figure 4, shows the highest priority concept-based assignment for a 
selected class. Here students can answer questions from in-class activities or homework assignments. 
Also depicted in Figure 4 is the student view of three question components: a multiple-choice ques¬ 
tion, a short answer written explanation, and a confidence selection. A sample (incorrect) student 
answer is also shown. 
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I 1187 Pure Species Phase 
1 Equilibrium - T 
1 1188 Pure Species Phase 
* Equilibrium - Ca 


✓ 95 8% 92-2,2 

V 75 8% 72-22.1 


96 Ans/96 Stu E 
95 Ans/96 Stu 5? 


| Original Throttling Valve 


y 20 6% 20-70,7 


1 * 1 - 



An ideal gas flows steadily through the piping system and valve shown 
pressure and temperature are Pi and Ti and the pressure drops through 
, a lower value. P 2 


Assuming the valve is well insulated and inlet and outlet pipes connected to the v 
are the same diameter, what is the relationship of the outlet temperature T; 
temperature Ti? 


i to the 

T 2 to t 


T 2 > Ti because work is 
done on the gas as it is 
compressed through the 
valve opening 
T 2 <Ti because 
temperature must 
decrease if pressure 
decreases since the 
volume and number of 
moles both stay the same 
T 2 = Ti because rapid 
expansion of an ideal gas ■ 
does not affect 
temperature 
Can't answer unless the 
type of gas flowing is 
specified 


1 Ideal gas - enthalpy 


y 62 2% 61-29,8 


Neutral 98Ans/97Stu H 


1 Ideal gas - enthalpy 


y 417% 40-40,16 


Neutral 96 Ans/97 Stu 


■ Orignal Throning Vafce 


and temperature are Pi and Ti and the pressive drops through the vaVe to a lower value Pi 
Assuming the valve is wel insulated and inlet and outlet pipes connected to the vatre are me 


done on the gas as rt is 

temperature must 
decrease it pressure 
decreases since the 


moles both stay the same 
Ij = T, because rapid 



Can t answer unless the 
type o< gas flowing is 
specified 


70 

20 

0 


Short Answer Exportation Word Cloud 


ideal Q3S pressure 
decrease 

tem pe ratU re energy .aw^wn^. 

volume constant drop an w dung* non 
Short Answer Explanation Word Cloud (by answer choice t 


Tj < Ti because temperaluie must decrease if 
iressure decreases since the vcaane and nrvnber of 
notes both stay the same 

idealgaSpressure 

decrease 

temperature*.,.™ 

motos vol U me constant law drop 

Tj = Ti because rapid expansion of an ideal gas 
floes not affect temperature 

ideal Q3S energy. 
temperature — 

fj > Ti because work is done on the gas as it is 

inlet ideal gas 

temperature 

pressure 



Figure 3. Statistics tab in the instructor interface of the AlChE Concept Warehouse 
showing question response data for a single class. The insets show more extensive 
information available for written responses. 


To obtain an account, students provide their email address and click the “log in” button, while 
leaving the password field empty. A link is then emailed which allows the students to create a 
password. The same process is used to reset forgotten passwords. Student email addresses and 
passwords are encrypted using a one-way hash; therefore password reminders cannot be sent. 
After an account is set up, students can log in for the first time and are greeted with a prompt to 
provide voluntary user information and an opportunity to provide informed consent to allow their 
anonymous responses to be incorporated into aggregate data for research purposes, as shown in 
the profile tab in Table 3 (Data Collection Principle). 

Technology use in the classroom, in the form of laptops (Efaw, Hampton, Martinez, & Smith, 2004; 
Griffin & Walker, 2005), tablet PCs (Cromack, 2008; Koile & Singer, 2006), and clickers (Duncan, 
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Tab 

Submenu Sections 

Description of Functions 

a 


Alerts students to the number of concept-based assignments with unanswered questions, 

a 


categorized by class. 

C/2 

G 

_o 


View the highest priority concept-based assignment for a selected class. Students can answer 

<D 

G 

o 


questions from in-class activities or homework assignments. 

qG 

Demographics 

Report voluntary demographic information (e.g., birth year, first year at the university, gender, 
race, and major). 

2 

PLh 

Informed consent 

Students can allow or deny the use of their response data for research purposes. 


Clicker registration 

Aligns clicker responses with the student’s account. 

Table 3. Summary of AlChE Concept Warehouse Student functions organized by tab and 

sub menu sections 


2005) has been well documented. The use of internet-capable cell phones is also emerging as a 
learning tool, seen in a recent study on active learning (Koretsky & Brooks, 2012) and as evidenced 
by audience response system companies creating cell phone applications as an alternative to pur¬ 
chasing separate hardware (elnstruction, 2013; i>cIicker, 2013; Inc., 2013; Turning Technologies, 2013). 
The student interface was designed with these considerations in mind. 

Material and Methods 

This mixed methods study aims at characterizing initial community use of the AlChE Concept 
Warehouse and determining faculty and student perceptions of the challenges of using the tool 
and the benefits it affords. 

Use Data 

Use data is available from the tool’s central database and has been analyzed to characterize the 
engagement of the chemical engineering faculty community over time. Data include identification 
of faculty users, their institutions, and how they became aware of the tool. The use statistics identify 
users both in the online and offline modes. For the online mode, the number of questions, number 
of students, and number of unique answers submitted are reported. For the offline mode, the only 
information available from the database is the number of questions downloaded. However, no informa¬ 
tion is available about how and the extent to which faculty use these questions, nor is there a record 
of faculty who use questions from the database, but retrieve them in other ways. Finally, use data is 
segregated by class type for the classes available: material balances, energy balances, thermodynam¬ 
ics, transport phenomena, chemical reaction engineering, materials science, chemistry, and physics. 
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AlChE CONCEPT WAHIflOUSE 


student@university.edu | logout | help 



Class: Thermodynamics ▼ 

Time remaining: 00:00:55 

An ideal gas flows steadily through the piping system and valve shown below. The inlet pressure and temperature are Pi and 
Ti and the pressure drops through the valve to a lower value, P 2 . 

Assuming the valve is well insulated and inlet and outlet pipes connected to the valve are the same diameter, what is the 
relationship of the outlet temperature T 2 to the inlet temperature T 1 ? 


Pi, Ti 


$4 


P 2 <Pi 

-► 


® T2 > Ti 
©T 2 = Ti 
© T 2 < Ti 

© Can't tell without Knowing the type of gas 


Please explain your answer in the box below 


since well insulated and no work is being done, deltaU has to equal zero. This 
means that the energy lost from the drop in pressure would lead to an increase in 
temperature. 


Please rate how confident you are with your answer 

substantially moderately neutral moderately substantially 
unsure unsure confident confident 

© © © @ © 


| Submit] 


Figure 4. Screenshot of a student’s (incorrect) answer from the questions tab from the 
student interface 


User Perceptions 
Faculty 

Ten faculty members who were early adopters were interviewed to investigate their perceptions 
of the AlChE Concept Warehouse. Three were beta testers that tested the website when it was in 
the early stages of development, seven had attended the Summer School Workshop where it was 
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made generally available to the community, and one heard about it from a colleague but did not 
attend the Workshop and was not a beta tester. One beta-tester also attended the workshop. 

Semi-structured interviews were conducted using an initial list of questions as a guide. Inter¬ 
view questions were designed around Rogers’ Theory of Diffusion of Innovations (Rogers, 2003). 
Interviews were conducted by two graduate student researchers. Participants were asked to 
provide IRB approved informed consent. Interviews averaged approximately 50 minutes. At the 
beginning of the interview, participants were told what the purpose of the interview was and that 
they should feel comfortable giving any comments or criticism. All interviews were audio recorded 
and transcribed. An emergent thematic coding process was performed on all transcripts (Braun 
& Clarke, 2006). 

Students 

A survey of student users was conducted for two cohorts in one of the classes in which the 
AlChE Concept Warehouse was used. The survey instrument consists of eight (1=strongly disagree 
to 5=strongly agree) Likert-scale statements and two questions which require written comments. 
It is described in detail elsewhere (Koretsky & Brooks, 2012). The students surveyed were in a 
junior-level class of approximately 100 students. While all students were asked to respond to the 
survey, only those who signed IRB approved informed consent forms were included in the analysis 
for this study. 

In this class, the tool was used in the online mode with students providing answers by laptops 
(mostly) and smart phones. Most students in the class had used the AlChE Concept Warehouse 
(or its predecessor) in at least two classes previously. Additionally, the class was taught by one of 
the primary developers of the tool. Thus, the responses reported in this study represent a specific 
case and are not intended to be comprehensive of the varied ways this tool is used throughout the 
community. 

To provide a richer view of students’ perception, the free response written comments were fur¬ 
ther analyzed. Questions focused on students’ technology problems and the perceived benefits, 
as follows: 

1. Describe any problems specifically based with technology that you encountered when the 
AlChE Concept Warehouse was used in class. 

2. Describe any benefits of using the AlChE Concept Warehouse in class. 

A coding process was used to identify common responses. The questions were coded by two 
graduate student researchers using an emergent process to develop categories. Once the catego¬ 
ries were identified, agreed upon and described, each researcher independently coded the student 
responses. The Cohen’s kappa was 0.98 indicating high reliability between researchers. 
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RESULTS AND DISCUSSION 


Preliminary Use 

The AlChE Concept Warehouse was made available to the chemical engineering community at the 
ASEE Chemical Engineering Summer School in July 2012. In the following year, over 300 faculty from 
over 100 institutions have registered for accounts. It has been actively used in both Online and Offline 
modes. In the online mode, over 2,800 questions have been answered by over 4,000 student users 
and over 125,000 answers have been submitted. For use offline, faculty have downloaded over 2,200 
questions. This number represents a lower bound since interviews show some instructors have copied 
questions directly from the interface rather than using the download feature. Although there is no way 
of telling how many individual questions have been answered, a conservative estimate scaling from 
online use would be well over 200,000 total from both modes. From these usage statistics it appears 
that the tool is rapidly being adopted and used by the chemical engineering academic community. 

Figure 5 shows the relative percentage of use by class type for questions asked online (Online 
Question %), answers submitted online (Online Answer %), and questions downloaded (Offline 



Figure 5. Plot of use percentage for different types of class available in the AlChE 
Concept Warehouse. Online use shows percentage of concept questions asked and 
percentage of student answers, whereas offline use shows percentage of questions 
downloaded. There have been no questions used online or downloaded offline for the class 
types “Reaction Engineering " and “Mass Transfer ” 
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Question %). For example, 27% of all the total online questions asked are identified as having concep¬ 
tual content relating to “Material Balances,” while 14% are “Energy Balances,” 12% “Thermodynamics,” 
etc. The proportion of answers submitted in material balances and energy balances is even greater 
(40% and 28% respectively). This result is reflective of the larger class size in the introductory course 
relative to those delivered later in the curriculum. 

These data reflect a user community early in its development and still evolving. While over half the 
questions that have been downloaded for offline use correspond to material balances, few energy 
balance questions have been downloaded. Almost all faculty who teach energy balances using the 
tool use it in the online mode. There also have been no questions used online or downloaded offline 
for the class types “Reaction Engineering” and “Mass Transfer.” For the former class type there are 
241 questions available, so the limited use is probably due to faculty awareness. On the other hand, 
for mass transfer only 10 questions exist, so the limitation is in question availability. 

To convey the progression of use, the number of faculty user accounts is plotted vs. time in 
Figure 6. The number of accounts of “active users” is shown for comparison. An active user is defined 



Figure 6. Plot of total and active accounts created since the release at 2012 ASEE Summer 
School 
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as someone who has logged in four or more times since the tool was made publically available in 
July 2012. Also included in the figure are lines marking major communication events that showed a 
large increase in faculty accounts. The greatest increase in accounts came as a result of the ASEE 
Summer School Workshop. Other large rises in the number of accounts resulted after the following 
communication activities: an email sent to chemical engineering (Chem E) and mechanical engineer¬ 
ing (ME) department heads that contained a promotional flyer to reach new adopters, two Virtual 
Communities of Practice (Thermodynamics and Mass and Energy Balances) meetings in which the 
AlChE Concept Warehouse was discussed and demonstrated, and following a presentation and 
demonstration of the AlChE Concept Warehouse at the 2013 ASEE Annual Conference. 

User Perceptions 
Faculty 

All ten faculty users who were interviewed noted positive design features of the AlChE Concept 
Warehouse website that afforded flexibility in implementation and use. Some participants sug¬ 
gested ways in which the user interface could be improved to be more familiar to other web-based 
interfaces they use and more user-friendly. Participants also noted the quality of questions as an 
important factor in their decision to use the tool. Some cited it as better than other resources they 
had previously used, while a couple noted hesitation and need to avoid “tricky” questions. In addi¬ 
tion, participants appreciated the large number of questions for most classes. 

One online adopter used the “display results” feature and described her/his implementation as 
follows: 

“I would go to class and then, what I’d try to do is lecture past that point, where I felt like I 
had explained something and my hope is that, okay now everyone is going to get it right, 
and then ask a question, a conceptual question, that maybe is phrased a little bit differently 
or might...even be perceived as being a little bit tricky, but if you understood the concept, 
it should be something you could get...and then challenge the students to answer it, and 
after a couple of minutes stopping it, showing the results, which were sometimes surprising 
for me as it was for the students...just that there was a question I thought everyone would 
get right and it would be 50/50, half the class would get it right, half the class would get it 
wrong... so then, we would use that as kind of a discussion point.” 

This example portrays how using a feature only available online gives instructors the immediate 
opportunity to refocus her/his lectures. This instructor also used the Concept Warehouse to admin¬ 
ister surveys and get immediate feedback from students, a use not foreseen by the developers. 
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The placement and number of ConcepTests administered differed between adopters. One adopter 
used them at the very beginning of class as a quiz on the previous reading assignment in an effort 
to motivate students to complete the assignment. Another adopter used ConcepTests throughout 
the class period and had a bank of questions available for use at the end of class, time permitting. 
Others had entire recitations dedicated to ConcepTests. Two adopters stated they used the Concept 
Warehouse more heavily towards the beginning of the term/semester and less as the term/semester 
went on due to time and/or difficulty of material. The number of ConcepTests used per day ranged 
from one to eight. Many of the adopters wrote their own questions and expressed interest in sharing 
them with other faculty. This interest aligns with the Community Contribution Principle. 

A more detailed analysis of faculty perception of the tool is available elsewhere (Gilbuena, Smith, 
Brooks, Finklestein, & Koretsky, 2013). 

Students 

Likert Survey Responses 

Responses to the Likert statements with a scale from 1 (strongly disagree) to 5 (strongly agree) 
are shown in Table 4. The first two items relate to perceived engagement, the third to self-efficacy, 
the fourth to seventh to perceived learning and metacognition, and the eighth to the usefulness 
of written reflection. In addition to showing values for the average response for each item, Table 4 
reports aggregated percent responses of “strongly disagree” and “disagree” as “negative response” 
and “agree” and “strongly agree” as “positive response.” 

Overall students responded positively with averages for engagement and learning items com¬ 
paring favorably to similar items reported for clickers (Han & Finkelstein, 2013). Students generally 
connected the use of the AlChE Concept Warehouse to learning with 155 of 179 students respond¬ 
ing positively to the statement that its use helped them to understand the concepts behind the 
problems and 145 of 181 students also agreeing that their conceptual understanding would in¬ 
crease in other classes if the tool were used in those classes. Statement 6 asks students to assess 
their awareness of misunderstandings while Statement 7 specifically connects such awareness to 
the tool. A chi-squared test shows that Statement 6 averages (4.08 and 4.00) are significantly 
greater (p < 10' 5 ) than Statement 7 averages (3.59 and 3.74). Interestingly, while students gener¬ 
ally responded very favorably both to the statement that they had to think more when they used 
the tool in class and the statement that the tool helped them understand the concepts behind the 
problems, they were less ready to attribute their awareness of misunderstandings specifically to 
the tool rather than the course. Such a result is consistent with the view that learning is promoted 
by the pedagogy and content the tool enables rather than the tool itself, as discussed by Beatty 
and Gerace (2009). 
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Statement 

Negative response 

Neutral 

response 

Positive 

response 

Average 

Year 1 

Year 2 

Year 1 

Year 2 

Year 1 

Year 2 

Year 1 

Year 2 

1. I am more actively involved in class 
when the AlChE Concept Warehouse 
is used 

5% 

8% 

20% 

15% 

78% 

76% 

4.01 

4.16 

2. I have to think more in class sessions 
that use the AlChE Concept Warehouse 
than those that do not. 

8% 

8% 

11% 

14% 

80% 

82% 

4.16 

4.18 

3. Seeing the class responses to a concept 
question (bar graph) helps increase my 
confidence. 

6% 

10% 

22% 

26% 

72% 

66% 

3.91 

3.87 

4. Using the AlChE Concept Warehouse 
helps me to understand the concepts 
behind the problems. 

2% 

7% 

14% 

4% 

84% 

89% 

4.14 

4.28 

5. If the AlChE Concept Warehouse was 
used in other classes, my conceptual 
understanding in those classes would 
be better. 

3% 

4% 

17% 

15% 

79% 

83% 

4.03 

4.02 

6. In this course, I am more aware of my 
misunderstandings than in courses 
taught by traditional methods. 

9% 

7% 

10% 

14% 

80% 

79% 

4.08 

4.00 

7. The change in awareness of my 

misunderstandings is due to the AlChE 
Concept Warehouse. 

13% 

8% 

28% 

24% 

60% 

68% 

3.59 

3.74 

8. The short answer follow-ups to multiple 
choice questions helped me to think 
more about the question and the answer 
that I chose. 

9% 

9% 

21% 

19% 

70% 

72% 

3.81 

3.85 


Table 4. Student responses to a subset of questions on the end of term survey. For 
Year 7, n=85 and Year 2, n=94. The responses of “strongly disagree” and “disagree” are 
aggregated into “negative response” and “agree” and “strongly agree” are aggregated 
into “positive response.” 


Free Response Questions 

In the sections below results and analysis from coding of students’ written responses to each 
free response question is presented. 

Technical Problems 

Figure 7 shows the number of responses to the categories for student responses to question 1: De¬ 
scribe any problems specifically based with technology that you encountered when the AlChE Concept 
Warehouse was used in class. The category “blank” indicates that the student left this item blank in 
responding to the survey. The most common response (n=63) was that there were no technical problems, 
e.g., “None, my phone worked great.” The other frequently coded category was internet connectivity 
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Figure 7. Frequency of coded responses for each category of technology problems 
question. 


(n=52). The majority of these responses indicated an anomalous event (“Only once was wireless down 
in the classroom, otherwise everything was good.”) while some responses alluded to more reoccurring 
problems (“Sometimes the internet wifi is slow.”). One such response in the latter group, indicated a 
perceived causal relationship between the technology problems and the student’s readiness to learn, 
“some times the internet was working really slow and i was stressing much about trying to get on the 
website and not spending time learning the content.” The remaining responses were approximately 
equally divided between technology problems with the specific devices the students were using (n=16), 
e.g., bringing laptop to class, hardware failure, other device problems, and with problems with navigat¬ 
ing the website software itself (n=20), e.g., submission of answers, log-in issues. 

Benefits 

We next discuss coding to question 2: Describe any benefits of using the AlChE Concept Ware¬ 
house in class. To interpret the responses, we use the model developed by Blasco-Arcas, Buil, 
HernaNdez-Ortega, & Sese (2013) to describe the benefits of the use of personal response systems 
in enhancing student learning performance. They find that “interactivity, active collaborative learning 
and engagement are three key underlying forces that explain the positive effects.” When students 
reflected on their learning experiences, common themes emerged that are consistent with this 
model, and illustrated by the following comments. 
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Regarding interactivity: 

• Opportunity to see what you don’t understand and what you do understand. Chance to have 
mistaken notions corrected. 

• It’s good to practice a lot of conceptual scenarios that we might not otherwise get to see, and 
then have a discussion about it. 

• The concept warehouse promotes problem solving and applying concepts in class instead of 
just listening to a professor lecture, which stimulates the understanding of the material. 

Regarding active collaborative learning: 

• Lets you actually apply some of the concepts that you learned in class, also it is nice when we 
talk to our classmates an(d) get a different view on the problem or a reinforcement of yours 

• (The) AlChE Concept Warehouse is a catalyst for conversation about challenging conceptual 
ideas. A lot of the concepts in thermodynamics are abstract and hard to understand because 
they conflict with our prior perceptions of the way the world works. By talking about them as 
a class and confronting them head on, we can better understand the concepts and correctly 
solve real life problems. 

Regarding Engagement: 

• It is more engaged than lecture. I get to see what other people said and their reasoning behind 
their answers which helps me to better understand why the answer is what it is. Rather than 
just knowing the answer. 

• (It) really helps to engage difficult concepts and have the professor explain them after we get 
a chance to chew on them 

• It engages me in class to start problem solving right away on the material we just covered. It let’s 
(sic) me work with others giving me different perspectives on how to solve the problem. 

In addition to interactivity, active collaborative learning and engagement, an additional common 
theme emerged from the coding process - general enjoyment during class when the tool was used. 
The following comments illustrate this theme: 

• I really enjoy coming to class when we use concept warehouse. It’s fun to test how well I un¬ 
derstand thermo throughout the course of the class. 

• it’s fun, like a game show that involves my learning about the subject 

• General enjoyment through the stronger feeling of participation. 

Technology Mediated Conceptual Learning 

The data above suggest that interactive use of the AlChE Concept Warehouse is perceived to 
enhance student engagement and learning of troublesome concepts. Coupled with findings reported 
from the literature, we infer that concept-based instruction is enabled by these types of tools. 
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Moreover, the rapid uptake of the AlChE Concept Warehouse by the chemical engineering com¬ 
munity indicates that the community is receptive of this type of resource and it can enable faculty 
to implement concept-based instruction more easily. 

However, as illustrated by the distribution of responses to the first free response question, imple¬ 
menting active learning pedagogies utilizing computers and other mobile devices in class presents 
challenges to students and instructors. It is not realistic to assume that the technology will be entirely 
transparent to students at all times. Inevitably, technology problems can arise for both instructors and 
students. When they occur, they consume the cognitive load needed to engage and learn trouble¬ 
some concepts. For this reason, technology problems can seem to magnify in perception. 

In terms of delivery in class, it is useful for instructors to be mentally ready to be adaptive with 
the day’s lesson plan. Even if technology problems are infrequent, students will take cues from 
how an instructor responds to them. Additionally, an important role in instruction becomes how 
the instructor frames the use of technology. Such framing should attempt to minimize students’ 
anxiety about the technical problems allowing them to focus on the conceptual mastery. Finally, as 
with any technological based resource, instructors should be aware of the learning curve for both 
themselves and their students. We encourage instructors not to abandon the use of technology too 
quickly during the initial “start-up” period of its use. 


CONCLUSION 

A web-based tool, the AlChE Concept Warehouse, has been developed to provide instructors a 
resource to help implement concept-based instruction. It has been well received by the community 
with hundreds of faculty joining in the first year of public release. Instructor and student perceptions 
are positive, viewing the tool as useful in increasing engagement and conceptual learning. While 
this tool specifically focuses on chemical engineering and related disciplines, the architecture of 
this tool is generic and not discipline specific. The following critical principles have been identified 
for design of such web-based tools: question quality, question quantity, emergent use, familiarity, 
support, community contribution, and data collection. 

This paper has focused on the development, initial community use, and faculty and student per¬ 
ceptions of the tool itself. However, we conclude with two broader questions that have emerged 
which warrant further investigation. What are the ways this type of tool can enable growth of a 
discipline-specific community of learning focused on concept-based instruction? In what ways does 
this type of tool provide empirical evidence of student learning that can enable education research¬ 
ers to help improve instruction and student learning? 
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